Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Port UFS-WM to Ursa #2471

Draft
wants to merge 14 commits into
base: develop
Choose a base branch
from
Draft

Port UFS-WM to Ursa #2471

wants to merge 14 commits into from
+180 −10

Conversation

ulmononian
Copy link
Collaborator

@ulmononian ulmononian commented Oct 15, 2024 •
edited
Loading

Commit Queue Requirements:

  • Fill out all sections of this template.
  • All sub component pull requests have been reviewed by their code managers.
  • Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
  • Commit 'test_changes.list' from previous step

Description:

Enable the UFS-WM to run on Ursa (Hera's follow-on machine). Currently, just using the pre-TDS via Niagara, so configurations will likely need to be updated as the machine comes closer to full implementation.

spack-stack installation to support UFS applications using Intel LLVM compilers is in progress; see JCSDA/spack-stack#1297.

UFS-WM_RT data will be staged on shared Niagara/Ursa disk space for now, until dedicated Ursa filesystem is made available; once the stack installation is finished, RTs will be run using 8 available nodes (1 service, 7 compute).

Commit Message:

* UFSWM - Port UFS-WM to Ursa

Priority:

  • Normal

Git Tracking

UFSWM:

Sub component Pull Requests:

  • None

UFSWM Blocking Dependencies:

  • None

Changes

Regression Test Changes (Please commit test_changes.list):

  • PR Adds New Tests/Baselines.

Input data Changes:

  • None (just staging on Ursa)

Library Changes/Upgrades:

  • Required

Testing Log:

  • RDHPCS
    • Hera
    • Orion
    • Hercules
    • Jet
    • Gaea
    • Derecho
  • WCOSS2
    • Dogwood/Cactus
    • Acorn
  • CI
  • opnReqTest (complete task if unnecessary)

@ulmononian
Copy link
Collaborator Author

ulmononian commented Nov 5, 2024 •
edited
Loading

quick update: this work is on hold while we wait for ursa's oso system, which will have internet connectivity and allow the spack-stack/1.8.0 installation to proceed. see JCSDA/spack-stack#1297 (comment).

@DusanJovic-NOAA DusanJovic-NOAA mentioned this pull request Dec 4, 2024
Closed
@ulmononian
Copy link
Collaborator Author

we are still waiting for internet connectivity to be enabled on ursa so that the spack-stack build can be completed. @RaghuReddy-NOAA do you have any update on this?

@RaghuReddy-NOAA
Copy link

@ulmononian The login node nfe91 is now able to access the external network. Please note that it is still behind a firewall, and any site that is reachable by Hera/Niagara should be reachable from nfe91 too.

@ulmononian
Copy link
Collaborator Author

ulmononian commented Jan 16, 2025 •
edited
Loading

currently testing ufs-wm w/ spack-stack/1.8.0 on ursa.

some issues with cmake and locating mpi libs right now, i.e., when trying to run cmake for the atm-only model

cmake -DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v17_p8,FV3_GFS_v17_p8_rrtmgp,FV3_GFS_v15_thompson_mynn_lam3km,FV3_WoFS_v0,FV3_GFS_v17_p8_mynn,FV3_GFS_v17_p8_ugwpv1 -D32BIT=ON ..

i get:

-- Could NOT find MPI_C (missing: MPI_C_WORKS)
CMake Error at /contrib/spack-stack/envs/1.8.0/ue-oneapi-ifort-2024.2.1/install/oneapi/2024.2.1/cmake-3.27.9-zfrh7no/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find MPI (missing: MPI_C_FOUND) (found version "3.1")
Call Stack (most recent call first):
  /contrib/spack-stack/envs/1.8.0/ue-oneapi-ifort-2024.2.1/install/oneapi/2024.2.1/cmake-3.27.9-zfrh7no/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /contrib/spack-stack/envs/1.8.0/ue-oneapi-ifort-2024.2.1/install/oneapi/2024.2.1/cmake-3.27.9-zfrh7no/share/cmake-3.27/Modules/FindMPI.cmake:1837 (find_package_handle_standard_args)
  CMakeLists.txt:148 (find_package)

this corresponds to an error generated during the find MPI section of the top-level ufs-wm CMakeLists.txt. below is the module show for the stack-intel-oneapi-mpi/2021.13 modulefile that is loaded at compile time:

[Cameron.Book@nfe91 build]$ module show  stack-intel-oneapi-mpi/2021.13
--------------------------------------------------------------------------------------------------------------------------------------------------------
   /contrib/spack-stack/envs/1.8.0/ue-oneapi-ifort-2024.2.1/install/modulefiles/oneapi/2024.2.1/stack-intel-oneapi-mpi/2021.13.lua:
--------------------------------------------------------------------------------------------------------------------------------------------------------
help([[]])
family("MetaMPI")
conflict("stack-intel-mpi")
conflict("stack-intel-oneapi-mpi")
conflict("stack-cray-mpich")
conflict("stack-mpich")
conflict("stack-mpt")
load("intel-oneapi-mpi/2021.13.1")
prereq("intel-oneapi-mpi/2021.13.1")
prepend_path("MODULEPATH","/contrib/spack-stack/envs/1.8.0/ue-oneapi-ifort-2024.2.1/install/modulefiles/intel-oneapi-mpi/2021.13-eaajhcw/oneapi/2024.2.1")
setenv("MPICC","mpiicx")
setenv("MPICXX","mpiicpx")
setenv("MPIF77","mpiifort")
setenv("MPIF90","mpiifort")
setenv("MPI_CC","mpiicx")
setenv("MPI_CXX","mpiicpx")
setenv("MPI_F77","mpiifort")
setenv("MPI_F90","mpiifort")
setenv("I_MPI_CC","mpiicc")
setenv("I_MPI_CXX","mpiicpc")
setenv("I_MPI_F77","/apps/spack-2024-12/linux-rocky9-x86_64/gcc-11.4.1/intel-oneapi-compilers-2024.2.1-oqhstbmawnrsdw472p4pjsopj547o6xs/compiler/2024.2/bin/ifort")
setenv("I_MPI_F90","mpiifort")
setenv("I_MPI_FC","mpiifort")
setenv("intel_oneapi_mpi_ROOT","/apps/spack-2024-12/linux-rocky9-x86_64/oneapi-2024.2.1/intel-oneapi-mpi-2021.13.1-ss72gbndvat3oz22sa6lhmlbjkeabrn4")
whatis("Name: stack-intel-oneapi-mpi")
whatis("Version: 2021.13")
whatis("Category: library")
whatis("Description: stack-intel-oneapi-mpi mpi library and module access")

i've also tried changing the envvars set in the ufs_ursa modulefile (mirroring some of the approaches in the hercules/hera/gaeac6 llvm lua files), to no avail so far. note that we are using icx,icpx, and ifort for now, though we do have an ifx-based stack as well. i've also tried w/ system cmake, but no difference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Porting to Ursa (Hera replacement)
2 participants
@ulmononian @RaghuReddy-NOAA